Constraints on decoding picture buffer

ABSTRACT

Methods, systems and devices for implementing constraints used in video encoding and video decoding are described. An example method of video processing includes performing a conversion between a video comprising one or more pictures comprising one or more slices and a bitstream of the video. The conversion conforms to a rule, and the bitstream is organized into one or more access units. The rule specifies a constraint on a number of decoded pictures stored in a decoded picture buffer (DPB). Each decoded picture of the decoded pictures is (i) marked as used for reference, (ii) has a flag indicative of the decoded picture being output, and (iii) has an output time later than a decoding time of a current picture.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of International Patent ApplicationNo. PCT/US2021/036469 filed on Jun. 8, 2021, which claims the priorityto and benefits of U.S. Provisional Patent Application No. 63/036,321filed on Jun. 8, 2020. All the aforementioned patent applications arehereby incorporated by reference in their entireties.

TECHNICAL FIELD

The present disclosure relates to image and video coding and decoding.

BACKGROUND

Digital video accounts for the largest bandwidth use on the internet andother digital communication networks. As the number of connected userdevices capable of receiving and displaying video increases, it isexpected that the bandwidth demand for digital video usage will continueto grow.

SUMMARY

The present disclosure discloses techniques that can implementconstraints used by video encoders and decoders to perform videoencoding or decoding.

In one example aspect, a video processing method is disclosed. Themethod includes performing a conversion between a video comprising oneor more pictures comprising one or more slices and a bitstream of thevideo, wherein the bitstream is organized into a plurality of accessunits (AUs), AU 0 to AU n, based on a format rule, where n is a positiveinteger, wherein the format rule specifies a relationship betweenremoval times of each of the plurality of AUs from a coded picturebuffer (CPB) during decoding and a number of slices in the each of theplurality of AUs.

In another example aspect, another video processing method is disclosed.The method includes performing a conversion between a video comprisingone or more pictures comprising one or more tiles and a bitstream of thevideo, wherein the bitstream is organized into a plurality of accessunits (AUs), AU 0 to AU n, based on a format rule, wherein n is apositive integer, wherein the format rule specifies a relationshipbetween removal times of each of the plurality of AUs from a codedpicture buffer (CPB) and a number of tiles in the each of the pluralityof AUs.

In yet another example aspect, another video processing method isdisclosed. The method includes performing a conversion between a videocomprising one or more pictures comprising one or more slices and abitstream of the video, wherein the conversion conforms to a rule,wherein the bitstream is organized into one or more access units,wherein the rule specifies a constraint on a number of decoded picturesstored in a decoded picture buffer (DPB), wherein each decoded pictureof the decoded pictures is (i) marked as used for reference, (ii) has aflag indicative of the decoded picture being output, and (iii) has anoutput time later than a decoding time of a current picture.

In yet another example aspect, a video encoder apparatus is disclosed.The video encoder comprises a processor configured to implementabove-described methods.

In yet another example aspect, a video decoder apparatus is disclosed.The video decoder comprises a processor configured to implementabove-described methods.

In yet another example aspect, a computer readable medium having codestored thereon is disclose. The code embodies one of the methodsdescribed herein in the form of processor-executable code.

These, and other, features are described throughout the presentdisclosure.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing an example video processing system inwhich various techniques disclosed herein may be implemented.

FIG. 2 is a block diagram of an example hardware platform used for videoprocessing.

FIG. 3 is a block diagram that illustrates an example video codingsystem that can implement some embodiments of the present disclosure.

FIG. 4 is a block diagram that illustrates an example of an encoder thatcan implement some embodiments of the present disclosure.

FIG. 5 is a block diagram that illustrates an example of a decoder thatcan implement some embodiments of the present disclosure.

FIGS. 6-8 show flowcharts for example methods of video processing.

DETAILED DESCRIPTION

Section headings are used in the present disclosure for ease ofunderstanding and do not limit the applicability of techniques andembodiments disclosed in each section only to that section. Furthermore,H.266 terminology is used in some description only for ease ofunderstanding and not for limiting scope of the disclosed techniques. Assuch, the techniques described herein are applicable to other videocodec protocols and designs also.

1. INTRODUCTION

This disclosure is related to video coding technologies. Specifically,it is about defining levels and bitstream conformance for a video codecthat supports both single-layer video coding and multi-layer videocoding. It may be applied to any video coding standard or non-standardvideo codec that supports single-layer video coding and multi-layervideo coding, e.g., Versatile Video Coding (VVC) that is beingdeveloped.

2. ABBREVIATIONS

-   -   APS Adaptation Parameter Set    -   AU Access Unit    -   AUD Access Unit Delimiter    -   AVC Advanced Video Coding    -   CLVS Coded Layer Video Sequence    -   CPB Coded Picture Buffer    -   CRA Clean Random Access    -   CTU Coding Tree Unit    -   CVS Coded Video Sequence    -   CLVSS Coded Layer Video Sequence Start    -   DPB Decoded Picture Buffer    -   DPS Decoding Parameter Set    -   DU Decoding Unit    -   EOB End Of Bitstream    -   EOS End Of Sequence    -   GCI General Constraints Information    -   GDR Gradual Decoding Refresh    -   HEVC High Efficiency Video Coding    -   HRD Hypothetical Reference Decoder    -   IDR Instantaneous Decoding Refresh    -   TRAP Intra Random Access Point    -   JEM Joint Exploration Model    -   MCTS Motion-Constrained Tile Sets    -   NAL Network Abstraction Layer    -   OLS Output Layer Set    -   PH Picture Header    -   PPS Picture Parameter Set    -   PTL Profile, Tier and Level    -   PU Picture Unit    -   RADL Random Access Decodable Leading    -   RASL Random Access Skipped Leading    -   RRP Reference Picture Resampling    -   RB SP Raw Byte Sequence Payload    -   SEI Supplemental Enhancement Information    -   SH Slice Header    -   SPS Sequence Parameter Set    -   STRP Short Term Reference Picture    -   SVC Scalable Video Coding    -   VCL Video Coding Layer    -   VPS Video Parameter Set    -   VTM VVC Test Model    -   VUI Video Usability Information    -   VVC Versatile Video Coding

3. INITIAL DISCUSSION

Video coding standards have evolved primarily through the development ofthe well-known International Telecommunication Union-TelecommunicationStandardization Sector (ITU-T) and International Organization forStandardization (ISO)/International Electrotechnical Commission (IEC)standards. The ITU-T produced H.261 and H.263, ISO/IEC produced MovingPicture Experts Group (MPEG)-1 and MPEG-4 Visual, and the twoorganizations jointly produced the H.262/MPEG-2 Video and H.264/MPEG-4Advanced Video Coding (AVC) and H.265/HEVC standards. Since H.262, thevideo coding standards are based on the hybrid video coding structurewherein temporal prediction plus transform coding are utilized. Toexplore the future video coding technologies beyond HEVC, the JointVideo Exploration Team (JVET) was founded by Video Coding Experts Group(VCEG) and MPEG jointly in 2015. Since then, many new methods have beenadopted by JVET and put into the reference software named JointExploration Model (JEM). The JVET meeting is concurrently held onceevery quarter, and the new coding standard is targeting at 50% bitratereduction as compared to HEVC. The new video coding standard wasofficially named as Versatile Video Coding (VVC) in the April 2018 JVETmeeting, and the first version of VVC test model (VTM) was released atthat time. As there are continuous effort contributing to VVCstandardization, new coding techniques are being adopted to the VVCstandard in every JVET meeting. The VVC working draft and test model VTMare then updated after every meeting. The VVC project is now aiming fortechnical completion (FDIS) at the July 2020 meeting.

3.1. Picture Resolution Change within a Sequence

In AVC and HEVC, the spatial resolution of pictures cannot change unlessa new sequence using a new SPS starts, with an TRAP picture. VVC enablespicture resolution change within a sequence at a position withoutencoding an TRAP picture, which is always intra-coded. This feature issometimes referred to as reference picture resampling (RPR), as thefeature needs resampling of a reference picture used for interprediction when that reference picture has a different resolution thanthe current picture being decoded.

The scaling ratio is restricted to be larger than or equal to 1/2 (2times downsampling from the reference picture to the current picture),and less than or equal to 8 (8 times upsampling). Three sets ofresampling filters with different frequency cutoffs are specified tohandle various scaling ratios between a reference picture and thecurrent picture. The three sets of resampling filters are appliedrespectively for the scaling ratio ranging from 1/2 to 1/1.75, from1/1.75 to 1/1.25, and from 1/1.25 to 8. Each set of resampling filtershas 16 phases for luma and 32 phases for chroma which is same to thecase of motion compensation interpolation filters. Actually the normalMC interpolation process is a special case of the resampling processwith scaling ratio ranging from 1/1.25 to 8. The horizontal and verticalscaling ratios are derived based on picture width and height, and theleft, right, top and bottom scaling offsets specified for the referencepicture and the current picture.

Other aspects of the VVC design for support of this feature that aredifferent from HEVC include: i) The picture resolution and thecorresponding conformance window are signaled in the PPS instead of inthe SPS, while in the SPS the maximum picture resolution is signaled.ii) For a single-layer bitstream, each picture store (a slot in the DPBfor storage of one decoded picture) occupies the buffer size as requiredfor storing a decoded picture having the maximum picture resolution.

3.2. Scalable Video Coding (SVC) in General and In VVC

Scalable video coding (SVC, sometimes also just referred to asscalability in video coding) refers to video coding in which a baselayer (BL), sometimes referred to as a reference layer (RL), and one ormore scalable enhancement layers (ELs) are used. In SVC, the base layercan carry video data with a base level of quality. The one or moreenhancement layers can carry additional video data to support, forexample, higher spatial, temporal, and/or signal-to-noise (SNR) levels.Enhancement layers may be defined relative to a previously encodedlayer. For example, a bottom layer may serve as a BL, while a top layermay serve as an EL. Middle layers may serve as either ELs or RLs, orboth. For example, a middle layer (e.g., a layer that is neither thelowest layer nor the highest layer) may be an EL for the layers belowthe middle layer, such as the base layer or any intervening enhancementlayers, and at the same time serve as a RL for one or more enhancementlayers above the middle layer. Similarly, in the Multiview or 3Dextension of the HEVC standard, there may be multiple views, andinformation of one view may be utilized to code (e.g., encode or decode)the information of another view (e.g., motion estimation, motion vectorprediction and/or other redundancies).

In SVC, the parameters used by the encoder or the decoder are groupedinto parameter sets based on the coding level (e.g., video-level,sequence-level, picture-level, slice level, etc.) in which they may beutilized. For example, parameters that may be utilized by one or morecoded video sequences of different layers in the bitstream may beincluded in a video parameter set (VPS), and parameters that areutilized by one or more pictures in a coded video sequence may beincluded in a sequence parameter set (SPS). Similarly, parameters thatare utilized by one or more slices in a picture may be included in apicture parameter set (PPS), and other parameters that are specific to asingle slice may be included in a slice header. Similarly, theindication of which parameter set(s) a particular layer is using at agiven time may be provided at various coding levels.

Thanks to the support of reference picture resampling (RPR) in VVC,support of a bitstream containing multiple layers, e.g., two layers withstandard definition (SD) and high definition (HD) resolutions in VVC canbe designed without the need any additional signal-processing-levelcoding tool, as upsampling needed for spatial scalability support canjust use the RPR upsampling filter. Nevertheless, high-level syntaxchanges (compared to not supporting scalability) are needed forscalability support. Scalability support is specified in VVC version 1.Different from the scalability supports in any earlier video codingstandards, including in extensions of AVC and HEVC, the design of VVCscalability has been made friendly to single-layer decoder designs asmuch as possible. The decoding capability for multi-layer bitstreams arespecified in a manner as if there were only a single layer in thebitstream. E.g., the decoding capability, such as DPB size, is specifiedin a manner that is independent of the number of layers in the bitstreamto be decoded. Basically, a decoder designed for single-layer bitstreamsdoes not need much change to be able to decode multi-layer bitstreams.Compared to the designs of multi-layer extensions of AVC and HEVC, thehigh level syntax (HLS) aspects have been significantly simplified atthe sacrifice of some flexibilities. For example, an IRAP AU is requiredto contain a picture for each of the layers present in the CVS.

3.3. Parameter Sets

AVC, HEVC, and VVC specify parameter sets. The types of parameter setsinclude SPS, PPS, APS, and VPS. SPS and PPS are supported in all of AVC,HEVC, and VVC. VPS was introduced since HEVC and is included in bothHEVC and VVC. APS was not included in AVC or HEVC but is included in thelatest VVC draft text.

SPS was designed to carry sequence-level header information, and PPS wasdesigned to carry infrequently changing picture-level headerinformation. With SPS and PPS, infrequently changing information neednot to be repeated for each sequence or picture, hence redundantsignaling of this information can be avoided. Furthermore, the use ofSPS and PPS enables out-of-band transmission of the important headerinformation, thus not only avoiding the need for redundant transmissionsbut also improving error resilience.

VPS was introduced for carrying sequence-level header information thatis common for all layers in multi-layer bitstreams.

APS was introduced for carrying such picture-level or slice-levelinformation that needs quite some bits to code, can be shared bymultiple pictures, and in a sequence there can be quite many differentvariations.

3.4. Profiles, Tier, and Levels

Video coding standards usually specify profiles and levels. Some videocoding standards also specify tiers, e.g., HEVC and the being-developedVVC.

Profiles, tiers, and levels specify restrictions on bitstreams and hencelimits on the capabilities needed to decode the bitstreams. Profiles,tiers and levels may also be used to indicate interoperability pointsbetween individual decoder implementations.

Each profile specifies a subset of algorithmic features and limits thatshall be supported by all decoders conforming to that profile. Note thatencoders are not required to make use of all coding tools or featuressupported in a profile, while decoders conforming to a profile arerequired to support all coding tools or features.

Each level of a tier specifies a set of limits on the values that may betaken by the bitstream syntax elements. The same set of tier and leveldefinitions is usually used with all profiles, but individualimplementations may support a different tier and within a tier adifferent level for each supported profile. For any given profile, alevel of a tier generally corresponds to a particular decoder processingload and memory capability.

Capabilities of video decoders conforming to a video codec specificationare specified in terms of the ability to decode video streams conformingto the constraints of profiles, tiers and levels specified in the videocodec specification. When expressing the capabilities of a decoder for aspecified profile, the tier and level supported for that profile shouldalso be expressed.

3.5. Existing VVC Tier and Level Definitions

In the latest VVC draft text in WET-50152-v5, the tier and leveldefinitions are as follows.

A.4.1 General Tier and Level Limits

For purposes of comparison of tier capabilities, the tier withgeneral_tier_flag equal to 0 (i.e., the Main tier) is considered to be alower tier than the tier with general_tier_flag equal to 1 (i.e., theHigh tier). For purposes of comparison of level capabilities, aparticular level of a specific tier is considered to be a lower levelthan some other level of the same tier when the value of thegeneral_level_idc or sublayer_level_idc[i] of the particular level isless than that of the other level.

The following is specified for expressing the constraints in this annex:

-   -   Let AU n be the n-th AU in decoding order, with the first AU        being AU 0 (i.e., the 0-th AU).    -   For an OLS with OLS index TargetOlsIdx, the variables        PicWidthMaxInSamplesY, PicHeightMaxInSamplesY, and        PicSizeMaxInSamplesY, and the applicable dpb_parameters( )        syntax structure are derived as follows:        -   If NumLayersInOls[TargetOlsIdx] is equal to 1,            PicWidthMaxInSamplesY is set equal to            sps_pic_width_max_in_luma_samples, PicHeightMaxInSamplesY is            set equal to sps_pic_height_max_in_luma_samples, and            PicSizeMaxInSamplesY is set equal to            PicWidthMaxInSamplesY*PicHeightMaxInSamplesY, where            sps_pic_width_max_in_luma_samples and            sps_pic_height_max_in_luma_samples are found in the SPS            referred to by the layer in the OLS, and the applicable            dpb_parameters( )) syntax structure is also found in that            SPS.        -   Otherwise (NumLayersInOls[TargetOlsIdx] is greater than 1),            PicWidthMaxInSamplesY is set equal to            vps_ols_dpb_pic_width[MultiLayerOlsIdx[TargetOlsIdx]],            PicHeightMaxInSamplesY is set equal to            vps_ols_dpb_pic_height[MultiLayerOlsIdx[TargetOlsIdx]],            PicSizeMaxInSamplesY is set equal to            PicWidthMaxInSamplesY*PicHeightMaxInSamplesY, and the            applicable dpb_parameters( ) syntax structure is identified            by vps_ols_dpb_params_idx[MultiLayerOlsIdx[TargetOlsIdx]]            found in the VPS.    -   When the specified level is not level 15.5, bitstreams        conforming to a profile at a specified tier and level shall obey        the following constraints for each bitstream conformance test as        specified in Annex C:        -   a) PicSizeMaxInSamplesY shall be less than or equal to            MaxLumaPs, where MaxLumaPs is specified in Table A.1.        -   b) The value of PicWidthMaxInSamplesY shall be less than or            equal to Sqrt(MaxLumaPs*8).        -   c) The value of PicHeightMaxInSamplesY shall be less than or            equal to Sqrt(MaxLumaPs*8).        -   d) For each referenced PPS, the value of NumTileColumns            shall be less than or equal to MaxTileCols and the value of            NumTileRows shall be less than or equal to MaxTileRows,            where MaxTileCols and MaxTileRows are specified in Table            A.1.        -   e) For the VCL HRD parameters, CpbSize[Htid][i] shall be            less than or equal to CpbVclFactor*MaxCPB for at least one            value of i in the range of 0 to hrd_cpb_cnt_minus1,            inclusive, where CpbSize[Htid][i] is specified in clause            7.4.6.3 based on parameters selected as specified in clause            C.1, CpbVclFactor is specified in Table A.3 and MaxCPB is            specified in Table A.1 in units of CpbVclFactor bits.        -   f) For the NAL HRD parameters, CpbSize[Htid][i] shall be            less than or equal to CpbNalFactor*MaxCPB for at least one            value of i in the range of 0 to hrd_cpb_cnt_minus1,            inclusive, where CpbSize[Htid][i] is specified in clause            7.4.6.3 based on parameters selected as specified in clause            C.1, CpbNalFactor is specified in Table A.3, and MaxCPB is            specified in Table A.1 in units of CpbNalFactor bits.

Table A.1 specifies the limits for each level of each tier for levelsother than level 15.5.

A tier and level to which a bitstream conforms are indicated by thesyntax elements general_tier_flag and general Jevel_idc, and a level towhich a sublayer representation conforms are indicated by the syntaxelement sublayer_level_idc[i], as follows:

-   -   If the specified level is not level 15.5, general_tier_flag        equal to 0 indicates conformance to the Main tier,        general_tier_flag equal to 1 indicates conformance to the High        tier, according to the tier constraints specified in Table A.1        and general_tier_flag shall be equal to 0 for levels below level        4 (corresponding to the entries in Table A.1 marked with “-”).        Otherwise (the specified level is level 15.5), it is a        requirement of bitstream conformance that general_tier_flag        shall be equal to 1 and the value 0 for general_tier_flag is        reserved for future use by ITU-T|ISO/IEC and decoders shall        ignore the value of general_tier_flag.    -   general_level_idc and sublayer_level_idc[i] shall be set equal        to a value of general_level_idc for the level number specified        in Table A.1.

TABLE A.1 General tier and level limits Max CPB size MaxCPB Max luma(CpbVclFactor or picture size CpbNalFactor bits) Max slices Max # of Max# of general_level_idc MaxLumaPs Main High per picture tile rows tilecolumns Level value* (samples) tier tier MaxSlicesPerPicture MaxTileRowsMaxTileCols 1.0 16   36 864   350 — 16 1 1 2.0 32   122 880  1 500 — 161 1 2.1 35   245 760  3 000 — 20 1 1 3.0 48   552 960  6 000 — 30 2 23.1 51   983 040 10 000 — 40 3 3 4.0 64 2 228 224 12 000  30 000 75 5 54.1 67 2 228 224 20 000  50 000 75 5 5 5.0 80 8 912 896 25 000 100 000200 11 10 5.1 83 8 912 896 40 000 160 000 200 11 10 5.2 86 8 912 896 60000 240 000 200 11 10 6.0 96 35 651 584  80 000 240 000 600 22 20 6.1 9935 651 584  120 000  480 000 600 22 20 6.2 102 35 651 584  180 000  800000 600 22 20 *For the level number in the form of major · minor, thevalue of general_level_idc for each of the above-lisetd levels is equalto major * 16 + minor * 3.A.4.2 Profile-Specific Level Limits

The following is specified for expressing the constraints in this annex:

-   -   Let the variable fR be set equal to 1÷300.

The variable HbrFactor is defined as follows:

-   -   If the bitstream is indicated to conform to the Main 10 profile        or the Main 4:4:4 10 profile, HbrFactor is set equal to 1.

The variable BrVclFactor, which represents the VCL bit rate scalefactor, is set equal to CpbVclFactor*HbrFactor.

The variable BrNalFactor, which represents the NAL bit rate scalefactor, is set equal to CpbNalFactor*HbrFactor.

The variable MinCr is set equal to MinCrBase*MinCrScaleFactor÷HbrFactor.

When the specified level is not level 15.5, the value ofmax_dec_pic_buffering_minus1[Htid]+1 shall be less than or equal toMaxDpbSize, which is derived as follows:

-   -   if(PicSizeMaxInSamplesY<=(MaxLumaPs>>2))        -   MaxDpbSize=Min(4*maxDpbPicBuf, 16)    -   else if(PicSizeMaxInSamplesY<=(MaxLumaPs>>1))        -   MaxDpbSize=Min(2*maxDpbPicBuf. 16) (A.1)    -   else ifX PicSizeMaxInSamplesY<=((3*MaxLumaPs)>>2))        -   MaxDpbSize=Min((4*maxDpbPicBuf)/3, 16)    -   else        -   MaxDpbSize=maxDpbPicBuf

where MaxLumaPs is specified in Table A.1, maxDpbPicBuf is equal to 8,and max_dec_pic_buffering_minus1[Htid] is found in or derived from theapplicable dpb_parameters( ) syntax structure.

Let numDecPics be the number of pictures in AU n. The variableAuSizeMaxInSamplesY[n] is set equal to PicSizeMaxInSamplesY*numDecPics.

Bitstreams conforming to the Main 10 or Main 4:4:4 10 profile at aspecified tier and level shall obey the following constraints for eachbitstream conformance test as specified in Annex C:

-   -   a) The nominal removal time of AU n (with n greater than 0) from        the CPB, as specified in clause C.2.3, shall satisfy the        constraint that AuNominalRemovalTime[n]−AuCpbRemovalTime[n−1] is        greater than or equal to Max(AuSizeMaxInSamplesY[n−1]÷MaxLumaSr,        fR), where MaxLumaSr is the value specified in Table A.2 that        applies to AU n−1.    -   b) The difference between consecutive output times of pictures        of different AUs from the DPB, as specified in clause C.3.3,        shall satisfy the constraint that DpbOutputInterval[n] is        greater than or equal to Max(AuSizeMaxInSamplesY[n]÷MaxLumaSr,        fR), where MaxLumaSr is the value specified in Table A.2 for AU        n, provided that AU n has a picture that is output and AU n is        not the last AU of the bitstream that has a picture that is        output.    -   c) The removal time of AU 0 shall satisfy the constraint that        the number of slices in each picture in AU 0 is less than or        equal to Min(Max(1,        MaxSlicesPerPicture*MaxLumaSr/MaxLumaPs*(AuCpbRemovalTime[0]−AuNominalRemovalTime[0])+MaxSlicesPerPicture*PicSizeMaxInSamplesY/MaxLumaPs),        MaxSlicesPerPicture), for the value of PicSizeMaxInSamplesY of        picture 0, where MaxSlicesPerPicture, MaxLumaPs and MaxLumaSr        are the values specified in Table A.1 and Table A.2,        respectively, that apply to AU 0.    -   d) The difference between consecutive CPB removal times of AUs n        and n−1 (with n greater than 0) shall satisfy the constraint        that the number of slices in each picture in AU n is less than        or equal to Min((Max(1,        MaxSlicesPerPicture*MaxLumaSr/MaxLumaPs*(AuCpbRemovalTime[n]−AuCpbRemovalTime[n−1])),        MaxSlicesPerPicture), where MaxSlicesPerPicture, MaxLumaPs and        MaxLumaSr are the values specified in Table A.1 and Table A.2        that apply to AU n.    -   e) For the VCL HRD parameters, BitRate[Htid][i] shall be less        than or equal to BrVclFactor*MaxBR for at least one value of i        in the range of 0 to hrd_cpb_cnt_minus1, inclusive, where        BitRate[Htid][i] is specified in clause 7.4.6.3 based on        parameters selected as specified in clause C.1 and MaxBR is        specified in Table A.2 in units of BrVclFactor bits/s.    -   f) For the NAL HRD parameters, BitRate[Htid][i] shall be less        than or equal to BrNalFactor*MaxBR for at least one value of i        in the range of 0 to hrd_cpb_cnt_minus1, inclusive, where        BitRate[Htid][i] is specified in clause 7.4.6.3 based on        parameters selected as specified in clause C.1 and MaxBR is        specified in Table A.2 in units of BrNalFactor bits/s.    -   g) The sum of the NumBytesInNalUnit variables for AU 0 shall be        less than or equal to        FormatCapabilityFactor*(Max(AuSizeMaxInSamplesY[0],        fR*MaxLumaSr)+MaxLumaSr*(AuCpbRemovalTime[0]−AuNominalRemovalTime[0]))        MinCr, where MaxLumaSr and FormatCapabilityFactor are the values        specified in Table A.2 and Table A.3, respectively, that apply        to AU 0.    -   h) The sum of the NumByteslnNalUnit variables for AU n (with n        greater than 0) shall be less than or equal to        FormatCapabilityFactor*MaxLumaSr*(AuCpbRemovalTime[n]−AuCpbRemovalTime[n−1])        MinCr, where MaxLumaSr and FormatCapabilityFactor are the values        specified in Table A.2 and Table A.3 respectively, that apply to        AU n.    -   i) The removal time of AU 0 shall satisfy the constraint that        the number of tiles in each picture in AU 0 is less than or        equal to Min(Max(1,        MaxTileCols*MaxTileRows*120*(AuCpbRemovalTime[0]−AuNominalRemovalTime[0])+MaxTileCols*MaxTileRows*AuSizeMaxInSamplesY[0]/MaxLumaPs),        MaxTileCols*MaxTileRows), where MaxTileCols and MaxTileRows are        the values specified in Table A.1 that apply to AU 0.    -   j) The difference between consecutive CPB removal times of AUs n        and n−1 (with n greater than 0) shall satisfy the constraint        that the number of tiles in each picture in AU n is less than or        equal to Min(Max(1,        MaxTileCols*MaxTileRows*120*(AuCpbRemovalTime[n]−AuCpbRemovalTime[n−1])),        MaxTileCols*MaxTileRows), where MaxTileCols and MaxTileRows are        the values specified in Table A.1 that apply to AU n.

TABLE A.2 Tier and level limits for the video profiles Max bit rate MinMaxBR compression Max luma (BrVclFactor or ratio sample rate BrNalFactorbits/s) MinCrBase MaxLumaSr Main High Main High Level (samples/sec) tiertier tier tier 1     552 960   128 — 2 2 2    3 686 400  1 500 — 2 2 2.1   7 372 800  3 000 — 2 2 3   16 588 800  6 000 — 2 2 3.1   33 177 60010 000 — 2 2 4   66 846 720 12 000  30 000 4 4 4.1   133 693 440 20 000 50 000 4 4 5   267 386 880 25 000 100 000 6 4 5.1   534 773 760 40 000160 000 8 4 5.2 1 069 547 520 60 000 240 000 8 4 6 1 069 547 520 60 000240 000 8 4 6.1 2 139 095 040 120 000  480 000 8 4 6.2 4 278 190 080 240000  800 000 8 4

TABLE A.3 Specification of CpbVclFactor, CpbNalFactor,FormatCapabilityFactor and MinCrScaleFactor Profile CpbVclFactorCpbNalFactor FormatCapabilityFactor MinCrScaleFactor Main 10 1 000 1 1001.875 1.00 Main 4:4:4 10 2 500 2 750 3.750 0.75 Main 10 Still Picture 1000 1 100 1.875 1.00 Main 4:4:4 10 Still Picture 2 500 2 750 3.750 0.75

3.6. Existing VVC Bitstream Conformance Definition

In the latest VVC draft text in NET-S0152-v5, the bitstream conformancedefinition is as follows.

A.4 Bitstream Conformance

A bitstream of coded data conforming to this Specification shall fulfilall requirements specified in this clause.

The bitstream shall be constructed according to the syntax, semanticsand constraints specified in this Specification outside of this annex.

The first coded picture in a bitstream shall be an IRAP picture (i.e.,an IDR picture or a CRA picture) or a GDR picture.

The bitstream is tested by the HRD for conformance as specified inclause C.1.

Let currPicLayerld be equal to the nuh_layer_id of the current picture.

For each current picture, let the variables maxPicOrderCnt andminPicOrderCnt be set equal to the maximum and the minimum,respectively, of the PicOrderCntVal values of the following pictureswith nuh_layer_id equal to currPicLayerld:

-   -   The current picture.    -   The previous picture in decoding order that has Temporand and        ph_non_ref_pic_flag both equal to 0 and is not a RASL or RADL        picture.    -   The STRPs referred to by all entries in RefPicList[0] and all        entries in RefPicList[1] of the current picture.    -   All pictures n that have PictureOutputFlag equal to 1,        AuCpbRemovalTime[n] less than AuCpbRemovalTime[currPic] and        DpbOutputTime[n] greater than or equal to        AuCpbRemovalTime[currPic], where currPic is the current picture.

All of the following conditions shall be fulfilled for each of thebitstream conformance tests:

-   -   1. For each AU n, with n greater than 0, associated with a BP        SEI message, let the variable deltaTime90k[n] be specified as        follows:        deltaTime90k[n]=90000*(AuNominalRemovalTime[n]−AuFinalArrivalTime[n−1])  (C.17)        -   The value of InitCpbRemovalDelay[Htid][ScIdx] is constrained            as follows:            -   If cbr_flag[ScIdx] is equal to 0, the following                condition shall be true:                InitCpbRemovalDelay[Htid][ScIdx]<=Ceil(deltaTime90k[n])  (C.18)            -   Otherwise (cbr_flag[ScIdx] is equal to 1), the following                condition shall be true:                Floor(deltaTime90k[n])<=InitCpbRemovalDelay[Htid][ScIdx]<=Ceil(deltaTime90k[n])  (C.19)                -   NOTE 1—The exact number of bits in the CPB at the                    removal time of each AU or DU may depend on which BP                    SEI message is selected to initialize the BIRD.                    Encoders must take this into account to ensure that                    all specified constraints must be obeyed regardless                    of which BP SEI message is selected to initialize                    the BIRD, as the BIRD may be initialized at any one                    of the BP SEI messages.    -   2. A CPB overflow is specified as the condition in which the        total number of bits in the CPB is greater than the CPB size.        The CPB shall never overflow.    -   3. When low_delay_hrd_flag[Htid] is equal to 0, the CPB shall        never underflow. A CPB underflow is specified as follows:        -   If DecodingUnitHrdFlag is equal to 0, a CPB underflow is            specified as the condition in which the nominal CPB removal            time of AU n AuNominalRemovalTime[n] is less than the final            CPB arrival time of AU n AuFinalArrivalTime[n] for at least            one value of n.        -   Otherwise (DecodingUnitHrdFlagis equal to 1), a CPB            underflow is specified as the condition in which the nominal            CPB removal time of DU m DuNominalRemovalTime[m] is less            than the final CPB arrival time of DU m            DuFinalArrivalTime[m] for at least one value of m.    -   4. When DecodingUnitHrdFlag is equal to 1,        low_delay_hrd_flag[Htid] is equal to 1 and the nominal removal        time of a DU m of AU n is less than the final CPB arrival time        of DU m (i.e., DuNominalRemovalTime[m]<DuFinalArrivalTime[m]),        the nominal removal time of AU n shall be less than the final        CPB arrival time of AU n (i.e.,        AuNominalRemovalTime[n]<AuFinalArrivalTime[n]).    -   5. The nominal removal times of AUs from the CPB (starting from        the second AU in decoding order) shall satisfy the constraints        on AuNominalRemovalTime[n] and AuCpbRemovalTime[n] expressed in        clauses A.4.1 and A.4.2.    -   6. For each current picture, after invocation of the process for        removal of pictures from the DPB as specified in clause C.3.2,        the number of decoded pictures in the DPB, including all        pictures n that are marked as “used for reference”, or that have        PictureOutputFlag equal to 1 and CpbRemovalTime[n] less than        CpbRemovalTime[currPic], where currPic is the current picture,        shall be less than or equal to        max_decpic_buffering_minus1[Htid].    -   7. All reference pictures shall be present in the DPB when        needed for prediction. Each picture that has PictureOutputFlag        equal to 1 shall be present in the DPB at its DPB output time        unless it is removed from the DPB before its output time by one        of the processes specified in clause C.3.    -   8. For each current picture that is not a CLVSS picture, the        value of maxPicOrderCnt−minPicOrderCnt shall be less than        MaxPicOrderCntLsb/2.    -   9. The value of DpbOutputInterval[n] as given by Equation C.16,        which is the difference between the output times of a picture        and the first picture following it in output order and having        PictureOutputFlag equal to 1, shall satisfy the constraint        expressed in clause A.4.1 for the profile, tier and level        specified in the bitstream using the decoding process specified        in clauses 2 through 9.    -   10. For each current picture, when        bp_du_cpb_params_inpictiming_seiflag is equal to 1, let        tmpCpbRemovalDelaySum be derived as follows:        tmpCpbRemovalDelaySum=0        for(i=0;i<pt_num_decoding_units_minus1;i++)        tmpCpbRemovalDelaySum+=pt_du_cpb_removal_delay_increment_minus1[i][Htid]+1  (C.20)        -   The value of ClockSubTick*tmpCpbRemovalDelaySum shall be            equal to the difference between the nominal CPB removal time            of the current AU and the nominal CPB removal time of the            first DU in the current AU in decoding order.    -   11. For any two pictures m and n in the same CVS, when        DpbOutputTime[m] is greater than DpbOutputTime[n], the        PicOrderCntVal of picture m shall be greater than the        PicOrderCntVal of picture n.        -   NOTE 2—All pictures of an earlier CVS in decoding order that            are output are output before any pictures of a later CVS in            decoding order. Within any particular CVS, the pictures that            are output are output in increasing PicOrderCntVal order.    -   12. The DPB output times derived for all pictures in any        particular AU shall be the same.

4. TECHNICAL PROBLEMS SOLVED BY DISCLOSED TECHNICAL SOLUTIONS

The existing VVC design for level definitions has the followingproblems:

-   -   1) The definitions of the two level limits on the relationship        between the CPB removal times for AU 0 and for AU n (n>0) and        the number of slices, i.e., items c and d in clause A.4.2        (Profile-specific level limits), are based on        MaxSlicesPerPicture and the maximum picture size. However,        MaxSlicesPerPicture is defined as a picture-level limit, while        the CPB removal times are AU-level parameters.    -   2) The definitions of the two level limits on the relationship        between the CPB removal times for AU 0 and for AU n (n>0) and        the number of tiles, i.e., items i and j in clause A.4.2        (Profile-specific level limits), are based on        MaxTileCols*MaxTileRows and the maximum AU size. However,        similarly as above, MaxTileCols and MaxTileRows are defined as a        picture-level limits, while the CPB removal times are AU-level        parameters.    -   3) The 6th constraint in cause C.4 (Bitstream conformance) is as        follows:        -   For each current picture, after invocation of the process            for removal of pictures from the DPB as specified in clause            C.3.2, the number of decoded pictures in the DPB, including            all pictures n that are marked as “used for reference”, or            that have PictureOutputFlag equal to 1 and CpbRemovalTime[n]            less than CpbRemovalTime[currPic], where currPic is the            current picture, shall be less than or equal to            max_decpic_buffering_minus1[Htid].    -    The part “CpbRemovalTime[n] less than CpbRemovalTime[currPic]”        describes a condition that the decoding time of a decoded        picture in the DPB is less than the decoding time of the current        picture. However, all decoded pictures in the DPB are always        decoded earlier than decoding of the current picture and thus        CpbRemovalTime[n] in the context is always less than        CpbRemovalTime[currPic].

5. EXAMPLES OF SOLUTIONS AND EMBODIMENTS

To solve the above problems, and others, methods as summarized below aredisclosed. The items should be considered as examples to explain thegeneral concepts and should not be interpreted in a narrow way.Furthermore, these items can be applied individually or combined in anymanner.

-   -   1) To solve the first problem, change the definitions of the two        level limits on the relationship between the CPB removal times        for AU 0 and for AU n (n>0) and the number of slices i.e., items        c and d in clause A.4.2 of the latest VVC draft, from being        based on MaxSlicesPerPicture and the maximum picture size to be        based on MaxSlicesPerPicture*(the number of pictures in the AU)        and the maximum AU size.    -   2) To solve the second problem, change the definitions of the        two level limits on the relationship between the CPB removal        times for AU 0 and for AU n (n>0) and the number of tiles i.e.,        items i and j in clause A.4.2 of the latest VVC draft, from        being based on MaxTileCols*MaxTileRows and the maximum AU size        to be based on MaxTileCols*MaxTileRows*(the number of pictures        in the AU) and the maximum AU size.    -   3) To solve the third problem, change the 6th constraint in        clause C.4 of the latest VVC draft to impose a constraint on the        number of decoded pictures stored in the DPB that are marked as        “used for reference”, have PictureOutputFlag equal to 1, and        have output time later than the decoding time of the current        picture.        -   a. In one example, in the 6th constraint in clause C.4 of            the latest VVC draft, change “or that have PictureOutputFlag            equal to 1 and CpbRemovalTime[n] less than            CpbRemovalTime[currPic]” to be “or have PictureOutputFlag            equal to 1 and DpbOutputTime[n] greater than            CpbRemovalTime[currPic]”.

6. EMBODIMENTS

Below are some example embodiments for some of the aspects summarizedabove in this Section, which can be applied to the VVC specification.The changed texts are based on the latest VVC text in WET-50152-v5. Mostrelevant parts that have been added or modified are bolded, underlinedand italicized, e.g., “using A and B”, and some of the deleted parts areitalicized and enclosed with double bolded brackets, e.g., “based on [[Aand]] B”. There may be some other changes that are editorial in natureand thus not highlighted.

6.1. Embodiment 1

This embodiment is for items 1 to 3 and their sub-items.

A.4.2 Profile-Specific Level Limits

The following is specified for expressing the constraints in this annex:

-   -   Let the variable fR be set equal to 1÷300.

The variable HbrFactor is defined as follows:

-   -   If the bitstream is indicated to conform to the Main 10 profile        or the Main 4:4:4 10 profile, HbrFactor is set equal to 1.

The variable BrVclFactor, which represents the VCL bit rate scalefactor, is set equal to CpbVclFactor*HbrFactor.

The variable BrNalFactor, which represents the NAL bit rate scalefactor, is set equal to CpbNalFactor*HbrFactor.

The variable MinCr is set equal to MinCrBase*MinCrScaleFactor÷HbrFactor.

When the specified level is not level 15.5, the value ofmax_dec_pic_buffering_minus1 [Htid]+1 shall be less than or equal toMaxDpbSize, which is derived as follows:

-   -   if(PicSizeMaxInSamplesY<=(MaxLumaPs>>2))        -   MaxDpbSize=Min(4*maxDpbPicBuf, 16)    -   else if(PicSizeMaxInSamplesY<=(MaxLumaPs>>1))        -   MaxDpbSize=Min(2*maxDpbPicBuf. 16) (A.1)    -   else ifX PicSizeMaxInSamplesY<=((3*MaxLumaPs)>>2))        -   MaxDpbSize=Min((4*maxDpbPicBuf)/3, 16)    -   else        -   MaxDpbSize=maxDpbPicBuf

where MaxLumaPs is specified in Table A.1, maxDpbPicBuf is equal to 8,and max_dec_pic_buffering_minus1[Htid] is found in or derived from theapplicable dpb_parameters( ) syntax structure.

Let numPics[n][[numDecPics]] be the number of pictures in AU n. Thevariable AuSizeMaxInSamplesY[n] is set equal toPicSizeMaxInSamplesY*numPics[n][[numDecPics]]. Bitstreams conforming tothe Main 10 or Main 4:4:4 10 profile at a specified tier and level shallobey the following constraints for each bitstream conformance test asspecified in Annex C:

-   -   k) The nominal removal time of AU n (with n greater than 0) from        the CPB, as specified in clause C.2.3, shall satisfy the        constraint that        -   AuNominalRemovalTime[n]−AuCpbRemovalTime[n−1] is greater            than or equal to Max(AuSizeMaxInSamplesY[n−1]÷MaxLumaSr,            fR), where MaxLumaSr is the value specified in Table A.2            that applies to AU n−1.    -   l) The difference between consecutive output times of pictures        of different AUs from the DPB, as specified in clause C.3.3,        shall satisfy the constraint that DpbOutputInterval[n] is        greater than or equal to Max(AuSizeMaxInSamplesY[n] MaxLumaSr,        fR), where MaxLumaSr is the value specified in Table A.2 for AU        n, provided that AU n has a picture that is output and AU n is        not the last AU of the bitstream that has a picture that is        output.    -   m) The removal time of AU 0 shall satisfy the constraint that        the number of slices in [[each picture in]]AU 0 is less than or        equal to Min(Max(1,        MaxSlicesPerPicture*numPics[0]*MaxLumaSr/MaxLumaPs*(AuCpbRemovalTime[0]−AuNominalRemovalTime[0])+MaxSlicesPerPicture*numPics[0]*AuSizeMaxInSamplesig        [0] [[PicSizeMaxInSamplesY]]/MaxLumaPs),        MaxSlicesPerPicture*numPics[0]), [[for the value of        PicSizeMaxInSamplesY of picture 0,]] where MaxSlicesPerPicture,        MaxLumaPs and MaxLumaSr are the values specified in Table A.1        and Table A.2, respectively, that apply to AU 0.    -   n) The difference between consecutive CPB removal times of AUs n        and n−1 (with n greater than 0) shall satisfy the constraint        that the number of slices in [[each picture in]] AU n is less        than or equal to Min((Max(1,        MaxSlicesPerPicture*numPics[n]1*MaxLumaSr/MaxLumaPs*(AuCpbRemovalTime[n]−AuCpbRemovalTime[n−1])),        MaxSlicesPerPicture*numPics[n 1], where MaxSlicesPerPicture,        MaxLumaPs and MaxLumaSr are the values specified in Table A.1        and Table A.2 that apply to AU n.    -   o) For the VCL HRD parameters, BitRate[Htid][i] shall be less        than or equal to BrVc1Factor*MaxBR for at least one value of i        in the range of 0 to hrd_cpb_cnt_minus1, inclusive, where        BitRate[Htid][i] is specified in clause 7.4.6.3 based on        parameters selected as specified in clause C.1 and MaxBR is        specified in Table A.2 in units of BrVclFactor bits/s.    -   p) For the NAL HRD parameters, BitRate[Htid][i] shall be less        than or equal to BrNalFactor*MaxBR for at least one value of i        in the range of 0 to hrd_cpb_cnt_minus1, inclusive, where        BitRate[Htid][i] is specified in clause 7.4.6.3 based on        parameters selected as specified in clause C.1 and MaxBR is        specified in Table A.2 in units of BrNalFactor bits/s.    -   q) The sum of the NumBytesInNalUnit variables for AU 0 shall be        less than or equal to        FormatCapabilityFactor*(Max(AuSizeMaxInSamplesY[0],        fR*MaxLumaSr)+MaxLumaSr*(AuCpbRemovalTime[0]−AuNominalRemovalTime[0]))÷MinCr,        where MaxLumaSr and FormatCapabilityFactor are the values        specified in Table A.2 and Table A.3, respectively, that apply        to AU 0.    -   r) The sum of the NumByteslnNalUnit variables for AU n (with n        greater than 0) shall be less than or equal to        FormatCapabilityFactor*MaxLumaSr*(AuCpbRemovalTime[n]−AuCpbRemovalTime[n−1])        MinCr, where MaxLumaSr and FormatCapabilityFactor are the values        specified in Table A.2 and Table A.3 respectively, that apply to        AU n.    -   s) The removal time of AU 0 shall satisfy the constraint that        the number of tiles [[in each picture]]in AU 0 is less than or        equal to Min(Max(1,        MaxTileCols*MaxTileRows*numPics[0]*120*(AuCpbRemovalTime[0]−AuNominalRemovalTime[0])+MaxTileCols*MaxTileRows*numPics[0]*AuSizeMaxInSamplesY[0]/MaxLumaPs),        MaxTileCols*MaxTileRows*numPics[0]), where MaxTileCols and        MaxTileRows are the values specified in Table A.1 that apply to        AU 0.    -   t) The difference between consecutive CPB removal times of AUs n        and n−1 (with n greater than 0) shall satisfy the constraint        that the number of tiles [[in each picture]]in AU n is less than        or equal to Min(Max(1,        MaxTileCols*MaxTileRows*numPics[n]*120*(AuCpbRemovalTime[n]−AuCpbRemovalTime[n−1])),        MaxTileCols*MaxTileRows*numPics[n]), where MaxTileCols and        MaxTileRows are the values specified in Table A.1 that apply to        AU n.        . . .        C.4 Bitstream Conformance        . . .    -   6. For each current picture, after invocation of the process for        removal of pictures from the DPB as specified in clause C.3.2,        the number of decoded pictures in the DPB, including all        pictures n that are marked as “used for reference”, or have        PictureOutputFlag equal to 1 and DpbOutputTime[n] greater than        CpbRemovalTime) [currPic] [[that have PictureOutputFlag equal to        1 and CpbRemovalTime[n] less than CpbRemovalTime[currPic]]],        where currPic is the current picture, shall be less than or        equal to max_dec_pic_buffering_minus1[Htid].

FIG. 1 is a block diagram showing an example video processing system1000 in which various techniques disclosed herein may be implemented.Various implementations may include some or all of the components of thesystem 1000. The system 1000 may include input 1002 for receiving videocontent. The video content may be received in a raw or uncompressedformat, e.g., 8 or 10 bit multi-component pixel values, or may be in acompressed or encoded format. The input 1002 may represent a networkinterface, a peripheral bus interface, or a storage interface. Examplesof network interface include wired interfaces such as Ethernet, passiveoptical network (PON), etc. and wireless interfaces such as wirelessfidelity (Wi-Fi) or cellular interfaces.

The system 1000 may include a coding component 1004 that may implementthe various coding or encoding methods described in the presentdisclosure. The coding component 1004 may reduce the average bitrate ofvideo from the input 1002 to the output of the coding component 1004 toproduce a coded representation of the video. The coding techniques aretherefore sometimes called video compression or video transcodingtechniques. The output of the coding component 1004 may be eitherstored, or transmitted via a communication connected, as represented bythe component 1006. The stored or communicated bitstream (or coded)representation of the video received at the input 1002 may be used bythe component 1008 for generating pixel values or displayable video thatis sent to a display interface 1010. The process of generatinguser-viewable video from the bitstream representation is sometimescalled video decompression. Furthermore, while certain video processingoperations are referred to as “coding” operations or tools, it will beappreciated that the coding tools or operations are used at an encoderand corresponding decoding tools or operations that reverse the resultsof the coding will be performed by a decoder.

Examples of a peripheral bus interface or a display interface mayinclude universal serial bus (USB) or high definition multimediainterface (HDMI) or Displayport, and so on. Examples of storageinterfaces include serial advanced technology attachment (SATA),peripheral component interconnect (PCI), integrated drive electronics(IDE) interface, and the like. The techniques described in the presentdisclosure may be embodied in various electronic devices such as mobilephones, laptops, smartphones or other devices that are capable ofperforming digital data processing and/or video display.

FIG. 2 is a block diagram of a video processing apparatus 2000. Theapparatus 2000 may be used to implement one or more of the methodsdescribed herein. The apparatus 2000 may be embodied in a smartphone,tablet, computer, Internet of Things (IoT) receiver, and so on. Theapparatus 2000 may include one or more processors 2002, one or morememories 2004 and video processing hardware 2006. The processor(s) 2002may be configured to implement one or more methods described in thepresent disclosure (e.g., in FIGS. 6-9 ). The memory (memories) 2004 maybe used for storing data and code used for implementing the methods andtechniques described herein. The video processing hardware 2006 may beused to implement, in hardware circuitry, some techniques described inthe present disclosure. In some embodiments, the hardware 2006 may bepartly or entirely in the one or more processors 2002, e.g., a graphicsprocessor.

FIG. 3 is a block diagram that illustrates an example video codingsystem 100 that may utilize the techniques of this disclosure. As shownin FIG. 3 , video coding system 100 may include a source device 110 anda destination device 120. Source device 110 generates encoded video datawhich may be referred to as a video encoding device. Destination device120 may decode the encoded video data generated by source device 110which may be referred to as a video decoding device. Source device 110may include a video source 112, a video encoder 114, and an input/output(I/O) interface 116.

Video source 112 may include a source such as a video capture device, aninterface to receive video data from a video content provider, and/or acomputer graphics system for generating video data, or a combination ofsuch sources. The video data may comprise one or more pictures. Videoencoder 114 encodes the video data from video source 112 to generate abitstream. The bitstream may include a sequence of bits that form acoded representation of the video data. The bitstream may include codedpictures and associated data. The coded picture is a codedrepresentation of a picture. The associated data may include sequenceparameter sets, picture parameter sets, and other syntax structures. I/Ointerface 116 may include a modulator/demodulator (modem) and/or atransmitter. The encoded video data may be transmitted directly todestination device 120 via I/O interface 116 through network 130 a. Theencoded video data may also be stored onto a storage medium/server 130 bfor access by destination device 120.

Destination device 120 may include an I/O interface 126, a video decoder124, and a display device 122.

I/O interface 126 may include a receiver and/or a modem. I/O interface126 may acquire encoded video data from the source device 110 or thestorage medium/server 130 b. Video decoder 124 may decode the encodedvideo data. Display device 122 may display the decoded video data to auser. Display device 122 may be integrated with the destination device120, or may be external to destination device 120 which be configured tointerface with an external display device.

Video encoder 114 and video decoder 124 may operate according to a videocompression standard, such as the High Efficiency Video Coding (HEVC)standard, Versatile Video Coding (VVC) standard and other current and/orfurther standards.

FIG. 4 is a block diagram illustrating an example of video encoder 200,which may be video encoder 114 in the system 100 illustrated in FIG. 3 .

Video encoder 200 may be configured to perform any or all of thetechniques of this disclosure. In the example of FIG. 4 , video encoder200 includes a plurality of functional components. The techniquesdescribed in this disclosure may be shared among the various componentsof video encoder 200. In some examples, a processor may be configured toperform any or all of the techniques described in this disclosure.

The functional components of video encoder 200 may include a partitionunit 201, a prediction unit 202 which may include a mode select unit203, a motion estimation unit 204, a motion compensation unit 205 and anintra prediction unit 206, a residual generation unit 207, a transformunit 208, a quantization unit 209, an inverse quantization unit 210, aninverse transform unit 211, a reconstruction unit 212, a buffer 213, andan entropy encoding unit 214.

In other examples, video encoder 200 may include more, fewer, ordifferent functional components. In an example, prediction unit 202 mayinclude an intra block copy (IBC) unit. The IBC unit may performprediction in an IBC mode in which at least one reference picture is apicture where the current video block is located.

Furthermore, some components, such as motion estimation unit 204 andmotion compensation unit 205 may be highly integrated, but arerepresented in the example of FIG. 4 separately for purposes ofexplanation.

Partition unit 201 may partition a picture into one or more videoblocks. Video encoder 200 and video decoder 300 may support variousvideo block sizes.

Mode select unit 203 may select one of the coding modes, intra or inter,e.g., based on error results, and provide the resulting intra- orinter-coded block to a residual generation unit 207 to generate residualblock data and to a reconstruction unit 212 to reconstruct the encodedblock for use as a reference picture. In some example, Mode select unit203 may select a combination of intra and inter prediction (CIIP) modein which the prediction is based on an inter prediction signal and anintra prediction signal. Mode select unit 203 may also select aresolution for a motion vector (e.g., a sub-pixel or integer pixelprecision) for the block in the case of inter-prediction.

To perform inter prediction on a current video block, motion estimationunit 204 may generate motion information for the current video block bycomparing one or more reference frames from buffer 213 to the currentvideo block. Motion compensation unit 205 may determine a predictedvideo block for the current video block based on the motion informationand decoded samples of pictures from buffer 213 other than the pictureassociated with the current video block.

Motion estimation unit 204 and motion compensation unit 205 may performdifferent operations for a current video block, for example, dependingon whether the current video block is in an I slice, a P slice, or a Bslice.

In some examples, motion estimation unit 204 may perform uni-directionalprediction for the current video block, and motion estimation unit 204may search reference pictures of list 0 or list 1 for a reference videoblock for the current video block. Motion estimation unit 204 may thengenerate a reference index that indicates the reference picture in list0 or list 1 that contains the reference video block and a motion vectorthat indicates a spatial displacement between the current video blockand the reference video block. Motion estimation unit 204 may output thereference index, a prediction direction indicator, and the motion vectoras the motion information of the current video block. Motioncompensation unit 205 may generate the predicted video block of thecurrent block based on the reference video block indicated by the motioninformation of the current video block.

In other examples, motion estimation unit 204 may perform bi-directionalprediction for the current video block, motion estimation unit 204 maysearch the reference pictures in list 0 for a reference video block forthe current video block and may also search the reference pictures inlist 1 for another reference video block for the current video block.Motion estimation unit 204 may then generate reference indexes thatindicate the reference pictures in list 0 and list 1 containing thereference video blocks and motion vectors that indicate spatialdisplacements between the reference video blocks and the current videoblock. Motion estimation unit 204 may output the reference indexes andthe motion vectors of the current video block as the motion informationof the current video block. Motion compensation unit 205 may generatethe predicted video block of the current video block based on thereference video blocks indicated by the motion information of thecurrent video block.

In some examples, motion estimation unit 204 may output a full set ofmotion information for decoding processing of a decoder.

In some examples, motion estimation unit 204 may not output a full setof motion information for the current video. Rather, motion estimationunit 204 may signal the motion information of the current video blockwith reference to the motion information of another video block. Forexample, motion estimation unit 204 may determine that the motioninformation of the current video block is sufficiently similar to themotion information of a neighboring video block.

In one example, motion estimation unit 204 may indicate, in a syntaxstructure associated with the current video block, a value thatindicates to the video decoder 300 that the current video block has thesame motion information as another video block.

In another example, motion estimation unit 204 may identify, in a syntaxstructure associated with the current video block, another video blockand a motion vector difference (MVD). The motion vector differenceindicates a difference between the motion vector of the current videoblock and the motion vector of the indicated video block. The videodecoder 300 may use the motion vector of the indicated video block andthe motion vector difference to determine the motion vector of thecurrent video block.

As discussed above, video encoder 200 may predictively signal the motionvector. Two examples of predictive signaling techniques that may beimplemented by video encoder 200 include advanced motion vectorprediction (AMVP) and merge mode signaling.

Intra prediction unit 206 may perform intra prediction on the currentvideo block. When intra prediction unit 206 performs intra prediction onthe current video block, intra prediction unit 206 may generateprediction data for the current video block based on decoded samples ofother video blocks in the same picture. The prediction data for thecurrent video block may include a predicted video block and varioussyntax elements.

Residual generation unit 207 may generate residual data for the currentvideo block by subtracting (e.g., indicated by the minus sign) thepredicted video block(s) of the current video block from the currentvideo block. The residual data of the current video block may includeresidual video blocks that correspond to different sample components ofthe samples in the current video block.

In other examples, there may be no residual data for the current videoblock for the current video block, for example in a skip mode, andresidual generation unit 207 may not perform the subtracting operation.

Transform processing unit 208 may generate one or more transformcoefficient video blocks for the current video block by applying one ormore transforms to a residual video block associated with the currentvideo block.

After transform processing unit 208 generates a transform coefficientvideo block associated with the current video block, quantization unit209 may quantize the transform coefficient video block associated withthe current video block based on one or more quantization parameter (QP)values associated with the current video block.

Inverse quantization unit 210 and inverse transform unit 211 may applyinverse quantization and inverse transforms to the transform coefficientvideo block, respectively, to reconstruct a residual video block fromthe transform coefficient video block. Reconstruction unit 212 may addthe reconstructed residual video block to corresponding samples from oneor more predicted video blocks generated by the prediction unit 202 toproduce a reconstructed video block associated with the current blockfor storage in the buffer 213.

After reconstruction unit 212 reconstructs the video block, loopfiltering operation may be performed reduce video blocking artifacts inthe video block.

Entropy encoding unit 214 may receive data from other functionalcomponents of the video encoder 200. When entropy encoding unit 214receives the data, entropy encoding unit 214 may perform one or moreentropy encoding operations to generate entropy encoded data and outputa bitstream that includes the entropy encoded data.

FIG. 5 is a block diagram illustrating an example of video decoder 300which may be video decoder 124 in the system 100 illustrated in FIG. 3 .

The video decoder 300 may be configured to perform any or all of thetechniques of this disclosure. In the example of FIG. 5 , the videodecoder 300 includes a plurality of functional components. Thetechniques described in this disclosure may be shared among the variouscomponents of the video decoder 300. In some examples, a processor maybe configured to perform any or all of the techniques described in thisdisclosure.

In the example of FIG. 5 , video decoder 300 includes an entropydecoding unit 301, a motion compensation unit 302, an intra predictionunit 303, an inverse quantization unit 304, an inverse transformationunit 305, and a reconstruction unit 306 and a buffer 307. Video decoder300 may, in some examples, perform a decoding pass generally reciprocalto the encoding pass described with respect to video encoder 200 (FIG. 4).

Entropy decoding unit 301 may retrieve an encoded bitstream. The encodedbitstream may include entropy coded video data (e.g., encoded blocks ofvideo data). Entropy decoding unit 301 may decode the entropy codedvideo data, and from the entropy decoded video data, motion compensationunit 302 may determine motion information including motion vectors,motion vector precision, reference picture list indexes, and othermotion information. Motion compensation unit 302 may, for example,determine such information by performing the AMVP and merge mode.

Motion compensation unit 302 may produce motion compensated blocks,possibly performing interpolation based on interpolation filters.Identifiers for interpolation filters to be used with sub-pixelprecision may be included in the syntax elements.

Motion compensation unit 302 may use interpolation filters as used byvideo encoder 20 during encoding of the video block to calculateinterpolated values for sub-integer pixels of a reference block. Motioncompensation unit 302 may determine the interpolation filters used byvideo encoder 200 according to received syntax information and use theinterpolation filters to produce predictive blocks.

Motion compensation unit 302 may use some of the syntax information todetermine sizes of blocks used to encode frame(s) and/or slice(s) of theencoded video sequence, partition information that describes how eachmacroblock of a picture of the encoded video sequence is partitioned,modes indicating how each partition is encoded, one or more referenceframes (and reference frame lists) for each inter-encoded block, andother information to decode the encoded video sequence.

Intra prediction unit 303 may use intra prediction modes for examplereceived in the bitstream to form a prediction block from spatiallyadjacent blocks. Inverse quantization unit 304 inverse quantizes, i.e.,de-quantizes, the quantized video block coefficients provided in thebitstream and decoded by entropy decoding unit 301. Inverse transformunit 305 applies an inverse transform.

Reconstruction unit 306 may sum the residual blocks with thecorresponding prediction blocks generated by motion compensation unit302 or intra-prediction unit 303 to form decoded blocks. If desired, adeblocking filter may also be applied to filter the decoded blocks inorder to remove blockiness artifacts. The decoded video blocks are thenstored in buffer 307, which provides reference blocks for subsequentmotion compensation/intra prediction and also produces decoded video forpresentation on a display device.

FIGS. 6-8 show example methods that can implement the technical solutiondescribed above in, for example, the embodiments show in FIGS. 1-5 .

FIG. 6 shows a flowchart for an example method 600 of video processing.The method 600 includes, at operation 610, performing a conversionbetween a video comprising one or more pictures comprising one or moreslices and a bitstream of the video, the bitstream being organized intoa plurality of access units (AUs), AU 0 to AU n, based on a format rulethat specifies a relationship between removal times of each of theplurality of AUs from a coded picture buffer (CPB) during decoding and anumber of slices in the each of the plurality of AUs, and n beingpositive integer.

FIG. 7 shows a flowchart for an example method 700 of video processing.The method 700 includes, at operation 710, performing a conversionbetween a video comprising one or more pictures comprising one or moretiles and a bitstream of the video, the bitstream being organized into aplurality of access units (AUs), AU 0 to AU n, based on a format rulethat specifies a relationship between removal times of each of theplurality of AUs from a coded picture buffer (CPB) and a number of tilesin the each of the plurality of AUs, and n being a positive integer.

FIG. 8 shows a flowchart for an example method 800 of video processing.The method 800 includes, at operation 810, performing a conversionbetween a video comprising one or more pictures comprising one or moreslices and a bitstream of the video, the bitstream being organized intoone or more access units, the conversion conforming to a rule thatspecifies a constraint on a number of decoded pictures stored in adecoded picture buffer, wherein each decoded picture of the decodedpictures is (i) marked as used for reference, (ii) has a flag indicativeof the decoded picture being output, and (iii) has an output time laterthan a decoding time of a current picture.

A listing of solutions preferred by some embodiments is provided next.

A1. A method of processing video data, comprising performing aconversion between a video comprising one or more pictures comprisingone or more slices and a bitstream of the video, wherein the bitstreamis organized into a plurality of access units (AUs), AU 0 to AU n, basedon a format rule, where n is a positive integer, wherein the format rulespecifies a relationship between removal times of each of the pluralityof AUs from a coded picture buffer (CPB) during decoding and a number ofslices in the each of the plurality of AUs.

A2. The method of solution A1, wherein the relationship is based on (i)a product of a maximum number of slices per picture and a number ofpictures in an access unit and (ii) a maximum size of the access unit.

A3. The method of solution A1, wherein the format rule specifies thatthe removal time of a first access unit, AU 0, of the plurality ofaccess units satisfies a constraint.

A4. The method of solution A3, wherein the constraint specifies that anumber of slices in AU 0 is less than or equal to Min(Max(1,MaxSlicesPerAu×MaxLumaSr/MaxLumaPs×(AuCpbRemovalTime[0]−AuNominalRemovalTime[0])+MaxSlicesPerAu×AuSizeMaxInSamplesY[0]/MaxLumaPs),MaxSlicesPerAu), wherein MaxSlicesPerAu is a maximum number of slicesper access unit, MaxLumaSr is a maximum luma sample rate, MaxLumaPs is amaximum luma picture size, AuCpbRemovalTime[m] is a CPB removal time ofan m-th access unit, AuNominalRemovalTime[m] is a nominal CPB removaltime of the m-th access unit, and AuSizeMaxInSamplesY[m] is a maximumsize, in luma samples, of a decoded picture that refers to a sequenceparameter set (SPS).

A5. The method of solution A4, wherein a value of MaxLumaPs and a valueof MaxLumaSr are selected from values corresponding to AU 0.

A6. The method of solution A1, wherein the format rule specifies that adifference between the removal times of two consecutive access units, AUn−1 and AU n, satisfies a constraint.

A7. The method of solution A6, wherein the constraint specifies that anumber of slices in AU n is less than or equal to Min((Max(1,MaxSlicesPerAu×MaxLumaSr/MaxLumaPs×(AuCpbRemovalTime[n]−AuCpbRemovalTime[n−1])),MaxSlicesPerAu), wherein MaxSlicesPerAu is a maximum number of slicesper access unit, MaxLumaSr is a maximum luma sample rate, MaxLumaPs is amaximum luma picture size, and AuCpbRemovalTime[m] is a CPB removal timeof an m-th access unit.

A8. The method of solution A7, wherein a value of MaxSlicesPerAu, avalue of MaxLumaPs, and a value of MaxLumaSr are selected from valuescorresponding to AU n.

A9. A method of processing video data, comprising performing aconversion between a video comprising one or more pictures comprisingone or more tiles and a bitstream of the video, wherein the bitstream isorganized into a plurality of access units (AUs), AU 0 to AU n, based ona format rule, wherein n is a positive integer, wherein the format rulespecifies a relationship between removal times of each of the pluralityof AUs from a coded picture buffer (CPB) and a number of tiles in theeach of the plurality of AUs.

A10. The method of solution A9, wherein the relationship is based on (i)a product of a maximum number of tile columns per picture (MaxTileCols),a maximum number of tile rows per picture (MaxTileRows), and a number ofpictures in an access unit and (ii) a maximum size of the access unit.

A11. The method of solution A9, wherein the format rule specifies thatthe removal time of a first access unit, AU 0, of the plurality ofaccess units satisfies a constraint.

A12. The method of solution A11, wherein the constraint specifies that anumber of tiles in AU 0 is less than or equal to Min(Max(1,MaxTilesPerAu×120×(AuCpbRemovalTime[0]−AuNominalRemovalTime[0])+MaxTilesPerAu×AuSizeMaxInSamplesY[0]/MaxLumaPs),MaxTilesPerAu), wherein MaxTilesPerAu is a maximum number of tiles peraccess unit, MaxLumaPs is a maximum luma picture size,AuCpbRemovalTime[m] is a CPB removal time of an m-th access unit,AuNominalRemovalTime[m] is a nominal CPB removal time of the m-th accessunit, and AuSizeMaxInSamplesY[m] is a maximum size, in luma samples, ofa decoded picture that refers to a sequence parameter set (SPS).

A13. The method of solution A12, wherein a value of MaxTilesPerAu isselected from values corresponding to AU 0.

A14. The method of solution A9, wherein the format rule specifies that adifference between the removal times of two consecutive access units, AUn−1 and AU n, satisfies a constraint.

A15. The method of solution A14, wherein the constraint specifies that anumber of tiles in AU n is less than or equal to Min(Max(1,MaxTilesPerAu×120×(AuCpbRemovalTime[n]−AuCpbRemovalTime[n−1])),MaxTilesPerAu), wherein MaxTilesPerAu is a maximum number of tiles peraccess unit and AuCpbRemovalTime[m] is a CPB removal time of an m-thaccess unit.

A16. The method of solution A15, wherein a value of MaxTilesPerAu isselected from values corresponding to AU n.

A17. The method of any of solutions A1 to A16, wherein the conversioncomprises decoding the video from the bitstream.

A18. The method of any of solutions A1 to A16, wherein the conversioncomprises encoding the video into the bitstream.

A19. A method of storing a bitstream representing a video to acomputer-readable recording medium, comprising generating the bitstreamfrom the video according to a method described in any one or more ofsolutions A1 to A16, and storing the bitstream in the computer-readablerecording medium.

A20. A video processing apparatus comprising a processor configured toimplement a method recited in any one or more of solutions A1 to A19.

A21. A computer-readable medium having instructions stored thereon, theinstructions, when executed, causing a processor to implement a methodrecited in one or more of solutions A1 to A19.

A22. A computer readable medium that stores the bitstream generatedaccording to any one or more of solutions A1 to 19.

A23. A video processing apparatus for storing a bitstream, wherein thevideo processing apparatus is configured to implement a method recitedin any one or more of solutions A1 to A19.

Another listing of solutions preferred by some embodiments is providednext.

B1. A method of processing video data, comprising performing aconversion between a video comprising one or more pictures comprisingone or more slices and a bitstream of the video, wherein the conversionconforms to a rule, wherein the bitstream is organized into one or moreaccess units, wherein the rule specifies a constraint on a number ofdecoded pictures stored in a decoded picture buffer (DPB), wherein eachdecoded picture of the decoded pictures is (i) marked as used forreference, (ii) has a flag indicative of the decoded picture beingoutput, and (iii) has an output time later than a decoding time of acurrent picture.

B2. The method of solution B1, wherein the number of decoded pictures isless than or equal to a maximum required size of the DPB in units ofpicture storage buffers minus 1.

B3. The method of solution B1, wherein the flag is PictureOutputFlag,the output time of an m-th picture is DpbOutputTime[m], and the decodingtime of the m-th picture is AuCpbRemovalTime[m].

B4. The method of any of solutions B1 to B3, wherein the conversioncomprises decoding the video from the bitstream.

B5. The method of any of solutions B1 to B3, wherein the conversioncomprises encoding the video into the bitstream.

B6. A method of storing a bitstream representing a video to acomputer-readable recording medium, comprising generating the bitstreamfrom the video according to a method described in any one or more ofsolutions B1 to B3, and storing the bitstream in the computer-readablerecording medium.

B7. A video processing apparatus comprising a processor configured toimplement a method recited in any one or more of solutions B1 to B6.

B8. A computer-readable medium having instructions stored thereon, theinstructions, when executed, causing a processor to implement a methodrecited in one or more of solutions B1 to B6.

B9. A computer readable medium that stores the bitstream generatedaccording to any one or more of solutions B1 to B6.

B10. A video processing apparatus for storing a bitstream, wherein thevideo processing apparatus is configured to implement a method recitedin any one or more of solutions B1 to B6.

Yet another listing of solutions preferred by some embodiments isprovided next.

P1. A video processing method, comprising performing a conversionbetween a video comprising one or more pictures comprising one or moreslices and a bitstream representation of the video, wherein thebitstream representation is organized into one or more access unitsaccording to a format rule, wherein the format rule specifies arelationship between one or more syntax elements in the bitstreamrepresentation and removal times for one or more access units from acoded picture buffer.

P2. The method of solution P1, wherein the rule specifies two levellimits for a relationship between the removal times based on a value ofa product of maximum number of slices per picture and a number ofpictures in the access unit, and a maximum allowed size of the accessunit.

P3. The method of solution P1, wherein the rule specifies two levellimits for a relationship between the removal times based on a firstparameter whose value is MaxTileCols*MaxTileRows*(a number of picturesin an access unit) and a second parameter whose value is a maximumallowed size of the one or more access units.

P4. A video processing method, comprising performing a conversionbetween a video comprising one or more pictures comprising one or moreslices and a bitstream representation of the video, wherein theconversion conforms to a rule wherein the bitstream representation isorganized into one or more access units, wherein the rule specifies aconstraint on a number of decoded pictures stored in a decoded picturebuffer that are marked as used for reference and have aPictureOutputFlag equal to 1, and have output time later than thedecoding time of the current picture.

P5. The method of any of solutions P1 to P4, wherein the performing theconversion comprises encoding the video to generate the codedrepresentation.

P6. The method of any of solutions P1 to P4, wherein the performing theconversion comprises parsing and decoding the coded representation togenerate the video.

P7. A video decoding apparatus comprising a processor configured toimplement a method recited in one or more of solutions P1 to P6.

P8. A video encoding apparatus comprising a processor configured toimplement a method recited in one or more of solutions P1 to P6.

P9. A computer program product having computer code stored thereon, thecode, when executed by a processor, causes the processor to implement amethod recited in any of solutions P1 to P6.

In the present disclosure, the term “video processing” may refer tovideo encoding, video decoding, video compression or videodecompression. For example, video compression algorithms may be appliedduring conversion from pixel representation of a video to acorresponding bitstream representation or vice versa. The bitstreamrepresentation (or simply, the bitstream) of a current video block may,for example, correspond to bits that are either co-located or spread indifferent places within the bitstream, as is defined by the syntax. Forexample, a macroblock may be encoded in terms of transformed and codederror residual values and also using bits in headers and other fields inthe bitstream.

The disclosed and other solutions, examples, embodiments, modules andthe functional operations described in this disclosure can beimplemented in digital electronic circuitry, or in computer software,firmware, or hardware, including the structures disclosed in thisdisclosure and their structural equivalents, or in combinations of oneor more of them. The disclosed and other embodiments can be implementedas one or more computer program products, i.e., one or more modules ofcomputer program instructions encoded on a computer readable medium forexecution by, or to control the operation of, data processing apparatus.The computer readable medium can be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter effecting a machine-readable propagated signal, or a combinationof one or more them. The term “data processing apparatus” encompassesall apparatus, devices, and machines for processing data, including byway of example a programmable processor, a computer, or multipleprocessors or computers. The apparatus can include, in addition tohardware, code that creates an execution environment for the computerprogram in question, e.g., code that constitutes processor firmware, aprotocol stack, a database management system, an operating system, or acombination of one or more of them. A propagated signal is anartificially generated signal, e.g., a machine-generated electrical,optical, or electromagnetic signal, that is generated to encodeinformation for transmission to suitable receiver apparatus.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a stand-alone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this disclosure can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., a field programmable gate array (FPGA) or anapplication specific integrated circuit (ASIC).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random-access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. However, a computerneed not have such devices. Computer readable media suitable for storingcomputer program instructions and data include all forms of non-volatilememory, media and memory devices, including by way of examplesemiconductor memory devices, e.g., erasable programmable read-onlymemory (EPROM), electrically erasable programmable read-only memory(EEPROM), and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto optical disks; and compact disc,read-only memory (CD ROM) and digital versatile disc read-only memory(DVD-ROM) disks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

While the present disclosure contains many specifics, these should notbe construed as limitations on the scope of any subject matter or ofwhat may be claimed, but rather as descriptions of features that may bespecific to particular embodiments of particular techniques. Certainfeatures that are described in the present disclosure in the context ofseparate embodiments can also be implemented in combination in a singleembodiment. Conversely, various features that are described in thecontext of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Moreover, the separation of various system components in theembodiments described in the present disclosure should not be understoodas requiring such separation in all embodiments.

Only a few implementations and examples are described and otherimplementations, enhancements and variations can be made based on whatis described and illustrated in the present disclosure.

What is claimed is:
 1. A method of processing video data, comprising: performing a conversion between a video comprising one or more pictures and a bitstream of the video, wherein the conversion conforms to a rule, wherein the bitstream is organized into one or more access units, wherein the rule specifies that a number of decoded pictures stored in a decoded picture buffer (DPB) is less than or equal to a maximum required size of the DPB in units of picture storage buffers minus 1 for each decoded picture (i) marked as used for reference or (ii) having a flag with a value indicative of the decoded picture being output and having an output time later than a decoding time of a current picture, and wherein the flag is PictureOutputFlag, the output time of an m-th picture is DpbOutputTime[m], and the decoding time of the m-th picture is AuCpbRemovalTime[m].
 2. The method of claim 1, wherein the output time is a DPB output time and the decoding time is a coded picture buffer (CPB) removal time.
 3. The method of claim 1, wherein the conversion comprises decoding the video from the bitstream.
 4. The method of claim 1, wherein the conversion comprises encoding the video into the bitstream.
 5. An apparatus for processing video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to: perform a conversion between a video comprising one or more pictures comprising one or more slices and a bitstream of the video, wherein the conversion conforms to a rule, wherein the bitstream is organized into one or more access units, wherein the rule specifies that a number of decoded pictures stored in a decoded picture buffer (DPB) is less than or equal to a maximum required size of the DPB in units of picture storage buffers minus 1 for each decoded picture (i) marked as used for reference or (ii) having a flag with a value indicative of the decoded picture being output and having an output time later than a decoding time of a current picture, and wherein the flag is PictureOutputFlag, the output time of an m-th picture is DpbOutputTime[m], and the decoding time of the m-th picture is AuCpbRemovalTime[m].
 6. The apparatus of claim 5, wherein the output time is a DPB output time and the decoding time is a coded picture buffer (CPB) removal time.
 7. The apparatus of claim 5, wherein the conversion comprises decoding the video from the bitstream.
 8. The apparatus of claim 5, wherein the conversion comprises encoding the video into the bitstream.
 9. A non-transitory computer-readable storage medium storing instructions that cause a processor to: perform a conversion between a video comprising one or more pictures comprising one or more slices and a bitstream of the video, wherein the conversion conforms to a rule, wherein the bitstream is organized into one or more access units, wherein the rule specifies that a number of decoded pictures stored in a decoded picture buffer (DPB) is less than or equal to a maximum required size of the DPB in units of picture storage buffers minus 1 for each decoded picture (i) marked as used for reference or (ii) having a flag with a value indicative of the decoded picture being output and having an output time later than a decoding time of a current picture, and wherein the flag is PictureOutputFlag, the output time of an m-th picture is DpbOutputTime[m], and the decoding time of the m-th picture is AuCpbRemovalTime[m].
 10. The storage medium of claim 9, wherein the output time is a DPB output time and the decoding time is a coded picture buffer (CPB) removal time.
 11. The storage medium of claim 9, wherein the conversion comprises decoding the video from the bitstream.
 12. The storage medium of claim 9, wherein the conversion comprises encoding the video into the bitstream.
 13. A non-transitory computer-readable recording medium storing a bitstream of a video which is generated by a method performed by a video processing apparatus, wherein the method comprises: generating the bitstream of the video comprising one or more pictures, wherein the generating of the bitstream conforms to a rule, wherein the bitstream is organized into one or more access units, wherein the rule specifies that a number of decoded pictures stored in a decoded picture buffer (DPB) is less than or equal to a maximum required size of the DPB in units of picture storage buffers minus 1 for each decoded picture (i) marked as used for reference or (ii) having a flag with a value indicative of the decoded picture being output and having an output time later than a decoding time of a current picture, and wherein the flag is PictureOutputFlag, the output time of an m-th picture is DpbOutputTime[m], and the decoding time of the m-th picture is AuCpbRemovalTime[m].
 14. The recording medium of claim 13, wherein the output time is a DPB output time and the decoding time is a coded picture buffer (CPB) removal time. 